Haptic effects with proximity sensing

ABSTRACT

A method of generating haptic effects on a device includes detecting the presence of an object near an input area of the device and generating a haptic effect on the device in response to the presence detection.

RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.11/744,636, filed on May 4, 2007, the specification of which is herebyincorporated by reference, which claims the benefit of U.S. ProvisionalPatent Application No. 60/828,195, filed Oct. 4, 2006.

FIELD OF THE INVENTION

One embodiment of the present invention is directed to a haptic feedbacksystem. More particularly, one embodiment of the present invention isdirected to a haptic feedback system that utilizes proximity sensing.

BACKGROUND INFORMATION

Electronic device manufacturers strive to produce a rich interface forusers. Conventional devices use visual and auditory cues to providefeedback to a user. In some interface devices, kinesthetic feedback(such as active and resistive force feedback) and/or tactile feedback(such as vibration, texture, and heat) is also provided to the user,more generally known collectively as “haptic feedback.” Haptic feedbackcan provide cues that enhance and simplify the user interface.Specifically, vibration effects, or vibrotactile haptic effects, may beuseful in providing cues to users of electronic devices to alert theuser to specific events, or provide realistic feedback to create greatersensory immersion within a simulated or virtual environment.

Haptic feedback has also been increasingly incorporated in portableelectronic devices, such as cellular telephones, personal digitalassistants (PDAs), portable gaming devices, and a variety of otherportable electronic devices. For example, some portable gamingapplications are capable of vibrating in a manner similar to controldevices (e.g., joysticks, etc.) used with larger-scale gaming systemsthat are configured to provide haptic feedback. Additionally, devicessuch as cellular telephones and PDAs are capable of providing variousalerts to users by way of vibrations. For example, a cellular telephonecan alert a user to an incoming telephone call by vibrating. Similarly,a PDA can alert a user to a scheduled calendar item or provide a userwith a reminder for a “to do” list item or calendar appointment.

Increasingly, portable devices are moving away from physical buttons infavor of touchscreen-only interfaces. This shift allows increasedflexibility, reduced parts count, and reduced dependence onfailure-prone mechanical buttons and is in line with emerging trends inproduct design. When using the touchscreen input device, a mechanicalconfirmation on button press or other user interface action can besimulated with haptics.

For portable devices, cost is an important driving factor. Therefore, togenerate haptic effects a single low cost actuator is generally used,for example an eccentric rotating mass (“ERM”) motor or anelectromagnetic motor. These actuators are able to produce strongmagnitude haptic outputs. However, they also require a certain amount oftime to achieve their peak haptic output (e.g., approximately 50 ms).These actuators are also used to provide feedback to the user whenoperating a touch sensitive input of a touchscreen device. For examplewhen the user presses a button on a touchscreen a haptic effect isoutput to give the sensation of pressing a mechanical button. It isdesired to output the haptic effect at the same time the user hasselected the button in the interface. However, due to the time it takesto have actuator reach a desired magnitude, the haptic effect lagsbehind the button press event. If this lag becomes too long the userwill not perceive the button press and the haptic effect as a singleevent.

Based on the foregoing, there is a need for an improved system andmethod for generating haptic effects for a touchscreen.

SUMMARY OF THE INVENTION

One embodiment is a method of generating haptic effects on a device. Themethod includes detecting the presence of an object near an input areaof the device and generating a haptic effect on the device in responseto the presence detection.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a cellular telephone in accordance with oneembodiment.

FIG. 2 is a flow diagram of the functionality performed by a telephonewhen generating haptic effects in response to the proximity of a user toa touchscreen in accordance with one embodiment.

FIG. 3 is a flow diagram of the functionality performed by the telephonewhen generating haptic effects in response to the proximity of a user tothe touchscreen in accordance with one embodiment.

DETAILED DESCRIPTION

One embodiment is a portable device that includes a haptic feedbacksystem with proximity sensing. The haptic system initiates the hapticfeedback before a user actually touches a touchscreen or other inputarea based on the proximity information.

FIG. 1 is a block diagram of a cellular telephone 10 in accordance withone embodiment. Telephone 10 includes a screen 11 and an input area,touchscreen 13, that includes “keys” and can include other types of userinterfaces, including menus, etc. In another embodiment, the keys oftelephone 10 can be mechanical type keys. Internal to telephone 10 is ahaptic feedback system that generates vibrations on telephone 10. In oneembodiment, the vibrations are generated on the entire telephone 10. Inother embodiments, specific portions of telephone 10 can be hapticallyenabled by the haptic feedback system, including the entire touchscreen13 or individual keys of touchscreen 13.

The haptic feedback system includes a processor 12. Coupled to processor12 is a memory 20 and an actuator drive circuit 16, which is coupled toa vibration actuator 18. Although the embodiment of FIG. 1 is a cellulartelephone, embodiments can be implemented with any type of handset ormobile/portable device, or any device that uses an actuator to generatevibrations. For example, other embodiments may not include touchscreensbut are haptic devices with other types of input areas. Other inputareas besides touchscreens may be a mini-joystick, scroll wheel, d-Pad,keyboard, touch sensitive surface, etc. As with a cellular telephone,for these devices there is a desire for a haptic effect to be generatedon the input area and/or the entire device.

Processor 12 may be any type of general purpose processor, or could be aprocessor specifically designed to provide haptic effects, such as anapplication-specific integrated circuit (“ASIC”). Processor 12 may bethe same processor that operates the entire telephone 10, or may be aseparate processor. Processor 12 can decide what haptic effects are tobe played and the order in which the effects are played based on highlevel parameters. In general, the high level parameters that define aparticular haptic effect include magnitude, frequency and duration. Lowlevel parameters such as streaming motor commands could also be used todetermine a particular haptic effect.

Processor 12 outputs the control signals to drive circuit 16 whichincludes electronic components and circuitry used to supply actuator 18with the required electrical current and voltage to cause the desiredhaptic effects. Actuator 18 is a haptic device that generates avibration on telephone 10. Actuator 18 can include one or more forceapplying mechanisms which are capable of applying a vibrotactile forceto a user of telephone 10 (e.g., via the housing of telephone 10).Memory device 20 can be any type of storage device, such as randomaccess memory (“RAM”) or read-only memory (“ROM”). Memory 20 storesinstructions executed by processor 12. Memory 20 may also be locatedinternal to processor 12, or any combination of internal and externalmemory.

A proximity sensor 14 is coupled to processor 12. Proximity sensor 14detects when a finger (or stylus) is in close proximity to but not incontact with touchscreen 13. Proximity sensor 14 may also detectlocation (e.g., x, y, z), direction, speed and acceleration, orientation(e.g., roll, pitch, yaw), etc. of the finger relative to touchscreen 13.Proximity sensor 14 provides its information as an input to processor 12when a finger is placed above touchscreen 13. This input can be used byprocessor 12 when generating haptic feedback for telephone 10.

Proximity sensor 14 may use any technology that allows the proximity ofa finger or other object to touchscreen 13 to be sensed. For example, itmay be based on sensing technologies including capacitive, electricfield, inductive, hall effect, reed, eddy current, magneto resistive,optical shadow, optical visual light, optical IR, optical colorrecognition, ultrasonic, acoustic emission, radar, heat, sonar,conductive or resistive and the like.

In one embodiment, proximity sensor 14 includes one or more proximitysensors that each generate a sensing field above touchscreen 13 and thatproduce signals when an object disturbs or intercepts the sensingfield(s). Each sensing field typically generates its own signals whendisturbed. In one embodiment, a single sensing field is used to coverthe entire touchscreen 13 surface. In another embodiment, a singlesensing field only covers a portion of the touchscreen 13 surface. Inanother embodiment, multiple sensing fields are used to cover the entiretouchscreen 13 surface. Any number of sensing fields may be used. Insome cases, in order to perform tracking, the sensing fields may even bedistributed as a pixelated array of nodes.

FIG. 2 is a flow diagram of the functionality performed by telephone 10when generating haptic effects in response to the proximity of a user totouchscreen 13 in accordance with one embodiment. In one embodiment, thefunctionality of FIG. 2 is implemented by software stored in memory andexecuted by a processor. In other embodiments, the functionality can beperformed by hardware, or any combination of hardware and software.

At 102, proximity sensor 14 senses the presence of a finger, stylus, orother object above or in some other manner near touchscreen 13 or otherinput area of telephone 10.

At 104, proximity sensor 14 determines the position, speed and/oracceleration of the finger relative to the surface of touchscreen 13.This enables processor 12 to determine whether the user's finger willactually contact touchscreen 13. For example, if the proximity signal isincreasing at a certain rate, it is highly likely that the user willcontact touchscreen 13 and press a button.

At 106, based on the determination at 104, processor 12 can calculatewhen the finger is expected to contact touchscreen 13. In anticipationof this contact, processor 12 initiates the haptic effect before theactual contact, thus avoiding the lag time caused by actuator 18.Processor 12 may use the acceleration of the finger and the startingtime required by actuator 18 to determine how far in advance to initiatethe haptic effect and energize actuator 18. Therefore, the haptic effectwill be implemented at approximately the exact time that the fingeractually contacts touchscreen 13 and result in better synchrony of thehaptic effect with the button press event. In another embodiment,processor 12 may initiate the haptic effect upon sensing the merepresence of the finger at 102.

In typical use of cell phones or PDA's, the user generally holds thedevice in one hand and uses the other hand to interact with the userinterface such as touchscreen 13. For handheld haptic devices withproximity sensing this means that the user can sense the haptic feedbackwith the hand holding the device even though the finger has not yettouched the surface. Therefore, useful haptic effects can be created asa function of the proximity even when a finger never touches touchscreen13.

In one embodiment, if a user is hovering a finger over the touchscreenand moving over a grid of displayed buttons, a first haptic effect canbe played when the user is moving from over one button to over the nextbutton. The first haptic effect can be a short soft haptic effect inorder to simulate the feel of moving over the edge of one button to thenext. This first haptic effect will give the user an indication of thebutton locations without the user activating the buttons. A secondhaptic effect can be played when the user actually touches the screenand acts to select the button. The second haptic effect can be a stronghaptic effect simulating a sharp button click.

FIG. 3 is a flow diagram of the functionality performed by telephone 10when generating haptic effects in response to the proximity of a user totouchscreen 13 in accordance with one embodiment. In one embodiment, thefunctionality of FIG. 3 is implemented by software stored in memory andexecuted by a processor. In other embodiments, the functionality can beperformed by hardware, or any combination of hardware and software.

At 202, proximity sensor 14 senses the presence and position of afinger, stylus, or other object above or in some other manner neartouchscreen 13. The sensed position may include the x and y coordinatesof the finger relative to touchscreen 13.

At 204, the functionality on the touchscreen is determined based onposition of the finger. For example, if a multiple button graphical userinterface on touchscreen 13 is displayed, the closest button that thefinger is above and the functionality of that button is determined.

At 206, processor 12 initiates a haptic effect based the functionalityand location of finger. Depending on the functionality on thetouchscreen, a different haptic effect may be generated by processor 12.Because the finger does not actually touch touchscreen 13 in thisembodiment, the haptic effect is felt by the other hand that is holdingtelephone 10.

In one embodiment, haptic effects can be applied to various menunavigation steps. For a user holding a hand held haptic enabled device,the haptics can be directed to the housing such that the user's handholding the device will experience the haptic effect. For example, whenusing a touchscreen, if the user hovers over a top level menu choice anda sub-menu exists for that choice, then a light haptic effect simulatinga soft button click (first haptic effect) can be played as the graphicsdisplay the preview sub-menu items. If there is no sub-menu attached tothat particular menu item (or no valid choices exist) then a secondhaptic effect can be generated, for example a buzz created by series ofpulses. If the user then selects the top level menu item, a stronghaptic effect simulating a sharp button click (third haptic effect) canbe played. If the user decides to move to another top-level menu afourth haptic effect can be played when entering the new menu. Thefourth haptic effect can be the same as the first haptic effect if it isa similar menu item. These haptic effects, resulting while a user fingeris still hovering over a surface, i.e., before contract has been made,would be directed to the other hand that is holding the haptic device.

In one embodiment, different haptic effects can be applied depending onthe intended use of the touchscreen area. For example, when a user'sfinger is hovering over a touch screen integrated into a cell phone orPDA, different haptic effects can be produced in the cell phone when ina map/GPS navigation mode than when managing climate control or whenentering hand written notes.

In one embodiment, during typical operation of a cell phone, the usercan have many different proximity relationships to the device. Forexample the cell phone could be in the user's hand, in the user'spocket, or right next to the user's face. These different operatingconditions required different levels of haptic feedback to the user. Aproximity sensor could be used to detect these different operatingconditions. It is also possible to detect these user conditions throughthe use of acceleration and/or orientation measurements. Once thecurrent operating condition of the device is known, for example holdingthe phone next to an ear, a global haptic volume adjustment could bemade for the different operating states. In one example, the user couldhave a specific vibration pattern set for a reminder event. If the userhas the phone attached to a belt clip the vibration pattern couldcontain strong haptic effects in order to signal the reminder event. Ifthe user is actually using the phone and has the phone pressed up nextto the user's face where the user will be more sensitive to the hapticeffects, when the reminder event is triggered, it would be useful tohave the haptic magnitude reduced for that event.

In one embodiment, gestures can be recognized such as a virtual jogdial. A jog dial recognizes rotation and when performed above aproximity active surface can be recognized without the surface beingtouched. A movement in the x, y and/or z direction can also berecognized. As an example, haptic effects can be attached to the use ofthe virtual jog wheel. A first haptic effect can be played at regularintervals of the jog wheel to give the user an indication andconfirmation of progress in moving the wheel. A second haptic effect canbe output when the user reaches the end of a list that the jog wheel iscontrolling. These jog and/or x, y, z motions can be recognized evenwhen performed with proximity sensing, over a mechanical alpha/numerickeypad such as which exists on most cell phones today.

Several embodiments disclosed are specifically illustrated and/ordescribed herein. However, it will be appreciated that modifications andvariations are covered by the above teachings and within the purview ofthe appended claims without departing from the spirit and intended scopeof the invention.

What is claimed is:
 1. A non-transitory computer readable medium havinginstructions stored thereon that, when executed by a processor, causethe processor to generate haptic effects on a device, the generatingcomprising: detecting a position of an object relative to a touchscreenof the device, wherein the detecting occurs while the object is hoveringover the touchscreen, and the device comprises a plurality of selectablefunctionality displayed on the touchscreen; based on the detectedposition, including the proximity of the hovering to one functionalityof the plurality of selectable functionality, determining afunctionality of the device; and generating a first haptic effect on thedevice while the object is hovering based at least on the determinedfunctionality.
 2. The computer readable medium of claim 1, furthercomprising: generating a user interface on the touchscreen that displaysa plurality of user interface elements providing the selectablefunctionality for the device.
 3. The computer readable medium of claim2, wherein the user interface elements comprise at least one of avirtual button, a virtual jog dial or a series of menus.
 4. The computerreadable medium of claim 1, wherein the first haptic effect is avibrotactile haptic effect and is generated on a housing of the device.5. The computer readable medium of claim 1, further comprising:detecting an operating condition of the device; generating a secondhaptic effect based at least on the detected operating condition.
 6. Thecomputer readable medium of claim 5, wherein the detected operatingcondition comprises a proximity to a user of the device.
 7. The computerreadable medium of claim 6, wherein the detected operating conditioncomprises the device is near a face of the user and wherein the secondhaptic effect comprises a reduced magnitude haptic effect.
 8. Thecomputer readable medium of claim 1, wherein the detected position isbased on a signal from one or more proximity sensors.
 9. The computerreadable medium of claim 1, wherein the first haptic effect is generatedwithout the object contacting the touchscreen.
 10. A method ofgenerating haptic effects on a device, the method comprising: detectinga position of an object relative to a touchscreen of the device, whereinthe detecting occurs while the object is not contacting the touchscreen,and the device comprises a plurality of selectable functionalitydisplayed on the touchscreen; based on the detected position, includingthe proximity of the object to one functionality of the plurality ofselectable functionality, determining a functionality of the device; andgenerating a first haptic effect on the device while the object is notcontacting the touchscreen based at least on the determinedfunctionality.
 11. The method of claim 10, further comprising:generating a user interface on the touchscreen that displays a pluralityof user interface elements providing the selectable functionality forthe device.
 12. The method of claim 11, wherein the user interfaceelements comprise at least one of a virtual button, a virtual jog dialor a series of menus.
 13. The method of claim 10, wherein the firsthaptic effect is a vibrotactile haptic effect and is generated on ahousing of the device.
 14. The method of claim 10, further comprising:detecting an operating condition of the device; generating a secondhaptic effect based at least on the detected operating condition. 15.The method of claim 14, wherein the detected operating conditioncomprises a proximity to a user of the device.
 16. The method of claim15, wherein the detected operating condition comprises the device isnear a face of the user and wherein the second haptic effect comprises areduced magnitude haptic effect.
 17. The method of claim 10, wherein thedetected position is based on a signal from one or more proximitysensors.
 18. A haptically enabled device comprising: a proximity sensor;a processor coupled to the proximity sensor; a haptic feedback systemcoupled to the processor; and a touchscreen coupled to the processor andadapted to display a plurality of selectable functionality; wherein theprocessor is adapted to: detect a position of an object relative to thetouchscreen based on a signal received from the proximity sensor,wherein the detecting occurs while the object is hovering over thetouchscreen; based on the detected position, including the proximity ofthe hovering to one functionality of the plurality of selectablefunctionality, determine a functionality of the device; and generate afirst haptic effect on the device while the object is hovering based atleast on the determined functionality.
 19. The device of claim 18, theprocessor further adapted to: generate a user interface on thetouchscreen that displays a plurality of user interface elementsproviding the selectable functionality for the device.
 20. The device ofclaim 19, wherein the user interface elements comprise at least one of avirtual button, a virtual jog dial or a series of menus.
 21. The deviceof claim 18, further comprising an actuator and coupled to the processorand a housing, wherein the first haptic effect is a vibrotactile hapticeffect and is generated by the actuator on the housing.
 22. The deviceof claim 18, the processor further adapted to: detect an operatingcondition of the device; generate a second haptic effect based at leaston the detected operating condition.
 23. The device of claim 22, whereinthe detected operating condition comprises a proximity to a user of thedevice.
 24. The device of claim 23, wherein the detected operatingcondition comprises the device is near a face of the user and whereinthe second haptic effect comprises a reduced magnitude haptic effect.